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Abstract 

Mission Operations and Command Assurance 
(MO&CA) is a Total Quality Management (TQM) 
task on JPL projects to instill quality in flight mission 
operations. From a system engineering view, 
MO&CA facilitates communication and problem- 
solving among flight teams and provides continuous 
process improvement to reduce risk in mission 
operations by addressing human factors. The 
MO&CA task has evolved from participating as a 
member of the spacecraft team, to an independent 
team reporting directly to flight project management 
and providing system level assurance. JPL flight 
projects have benefited significantly from MO&CA's 
effort to contain risk and prevent rather than rework 
errors. MO&CA’s ability to provide direct transfer of 
knowledge allows new projects to benefit from 
previous and ongoing flight experience. 
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1, Introduction 

A long-term program is in progress at the Jet 
Propulsion Laboratory (JPL) to reduce cost and risk 
of flight mission operations through defect prevention 
and error management. Flight mission operations 
require systems that place human operators in a 
demanding, high risk environment. This applies not 
only to mission controllers working in the "dark 
room” and Deep Space Network (DSN) operators 
configuring and monitoring DSN operations, but also 
to flight teams that plan the mission and develop the 
command sequences and to engineering teams 
responsible for analyzing spacecraft performance. 
The flight operations environment generally requires 
operators to make rapid, critical decisions and solve 
problems based on limited information, while closely 
following standard procedures (Refs. 1-3). This 
environment is, therefore, inherently risky because 
each decision made is potentially mission critical. 

To contain this risk at JPL, flight mission operations 
procedures (as described in Refs. 4-5) currently 


require intensive human reviews. In addition, when 
an error does occur, rapid rework is required to 
ensure mission success. This strategy has worked 
well to reduce risk and ensure the success of JPL 
missions. However, the large human labor 
investment required for review and rework has 
substantially contributed to the overall cost of flight 
mission operations and has placed operators in 
stressful environments. Prevention of errors would 
greatly reduce both cost and risk of flight projects. 
Thus, the motivation of the long-term defect 
prevention/error management program is to contain 
risk in a more cost effective and human supportive 
manner by preventing errors rather than reworking 
them. The goal of this program is the management, 
reduction and prevention of errors. 

A major element of this program is the Mission 
Operations and Command Assurance (MO&CA) 
function. MO&CA provides a system level function 
on flight projects to instill quality in flight mission 
operations. MO&CA's primary goal is to help 
improve the operational reliability of projects during 
flight. MO&CA occupies a unique position in the 
flight project organization, reporting to both flight 
project management and the Systems Assurance 
Division of the JPL Office of Engineering and 
Review. As a result, MO&CA is able to cross 
operational boundaries between teams and offices on 
a flight project enhancing inter-team communication 
and facilitating problem solving within the project. 

This paper describes the development and evolution 
of the MO&CA function at JPL and the benefits 
provided to flight projects by MO&CA. 

2 . Evolution of the MO&CA Task 

The MO&CA task began on the Voyager (VGR) 
project in 1985. In response to an increase in 
command related problems a study was conducted by 
the JPL Office of Engineering and Review to analyze 
the adequacy of procedures, operations and software 
involved in real-time commanding with the goal of 
reducing errors. Incident Surprise Anomaly (ISA) 
reports, problem reports written by flight team 
members when an anomaly occurs in flight 
operations, were analyzed from an eight year period 
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Voyager ISA Analysis - Command Error Causes (1977-1985) 


Figure 1 

Voyager ISA Analysis 


(1977 - 1985) to determine the causes of command 
errors. This study showed that the major cause for 
real-time command errors was human error (Figure 

1 ). 

Based on this analysis, recommendations were made 
to the VGR project for improvements including: 1) 
upgrading the command development software to 
improve readability of command printouts thereby 
facilitating command reviews and approvals; 2) 
providing traceability between command forms and 
ISAs to facilitate analysis and correction of command 
incidents; 3) reducing real-time commanding by 
improving the coordination of real-time and sequence 
commanding and including as many commands as 
possible in command sequences; and 4) updating 
flight team training to include command awareness 
issues to inform flight team members of potential 
command problems and how to avoid them. 
Command development procedures were updated to 
incorporate these recommendations. Real-time 
command anomalies decreased from 60 in 1985 to 40 
in 1986, and to 24 in both 1987 and 1988. 

When an opening occurred in the spacecraft team, the 
position was filled by a MO&CA engineer who 
became the Systems Lead for real-time commanding 
for the VGR Project. This placement allowed 


MO&CA to not only recommend changes to 
command procedures, but also to implement these 
changes with project management concurrence. 
MO&CA also continued to analyze ISA reports and 
make recommendations for continuous improvement 
to the commanding process. MO&CA provided both 
a system engineering function for the spacecraft team 
and a systems assurance function for the VGR 
Project. 

Following VGR, a MO&CA team was activated on 
the Magellan (MGN) Project in March 1989, just two 
months prior to launch. The main MO&CA task for 
MGN was to detect, analyze and correct defects that 
existed in flight operations and procedures. One of 
the major efforts of the MGN MO&CA team was 
assisting the flight project to upgrade the real-time 
command process and related operational procedures. 
The initial real-time command process in place at 
launch included only a handful of steps. Systems 
coordination and inter-team communication were not 
included in the procedures. 

In the first few months following launch extensive 
operational workarounds and real-time commanding 
were required to compensate for the spacecraft 
hardware problems. Because of the level of 
commanding and a lack of coordination in the real- 
time command process, command problems occurred. 
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MO&CA recommended improvements to the 
command process which included: 1) review of 
commands by all subsystems prior to development; 
2) system level coordination of all commanding; 3) 
management approval prior to command 
development; 4) traceability of commands from 
initial request to final approval for transmission; 5) 
development of rigid test requirements for all 
commands; 6) required representation by all 
operations teams at command review and approval 
meetings; 7) spacecraft team support of the command 
coordination meeting and 8) training for all flight 
team members with the newly developed command 
procedures. 

By December 1989, an updated real-time command 
process was in place on MGN. Improved 
communications enabled the flight team to function 
well as a unit and respond quickly to anomalies. 
Real-time command incidents decreased dramatically 
despite the fact that the flight team continued to face 
spacecraft anomalies. 

In contrast to the VGR project, the MGN team was 
not integrated directly into an existing team on the 
flight project, but instead formed an independent unit. 
While this enabled the MGN MO&CA team to 
maintain a systems view of flight operations, it did 
not provide the same ability to implement changes. 
MGN MO&CA instead provided recommendations 
for change based on ISA analysis and direct 
participation in working, review and approval 
meetings. The flight team, directed by project 
management, implemented the changes to operations 
procedures and processes. 

Due to the success of the VGR and MGN MO&CA 
teams, MO&CA teams were placed on the Mars 
Observer and TOPEX/POSEIDON projects. Both of 
these new projects experienced immediate benefits 
through the direct transfer of MO&CA's knowledge 
and experience from the previous two projects. 
These MO&CA teams were the first to be in place on 
the flight project an extended time prior to launch. 
The teams were therefore able to implement "lessons 
learned" and process improvements early. This 
opportunity allowed MO&CA to instill quality into 
the flight procedures in a pro-active manner, rather 
than work reactively to update processes and 
procedures after completion of mission operations 
development 

Mars Observer MO&CA, like the MGN MO&CA 
team, was established as an independent unit making 
recommendations for improvements and updates to 
command processes and procedures. A prime target 
for improvement by the Mars Observer MO&CA 
team was operations communications. 


The Mars Observer project had strong real-time and 
sequence commanding processes in place when 
MO&CA began working with the project. MO&CA, 
however, noted problems with inter-team 
communication and use of ancillary command data. 
Four separate operations teams. Spacecraft, Planning 
and Sequencing, Mission Control, and MO&CA, 
maintained separate command related data files that 
resulted in redundant and incongruous data. Manual 
transcription and interpretation errors occurred 
frequently and unnecessarily increased risk. 
MO&CA gathered the file structures and reports from 
each team and identified redundant data usage. 
MO&CA also initiated and led a working group that 
analyzed each team's data needs and identified and 
prioritized requirements for the development of a 
single command data system. The working group 
passed recommendations for system implementation 
to the Uplink Manager. An effort is currently 
underway to implement an on-line, real-time 
command data system to be in place by August 1993. 

Another communications issue that Mars Observer 
MO&CA addressed was the result of an unique 
aspect of the Mars Observer project. The principle 
investigators have direct control of commanding the 
science instruments. The remote science teams are 
situated at several different locations throughout the 
United States. Therefore, maintaining 
communications between the science teams and the 
flight operation team located at JPL is a challenge. 
Also, the science teams need to have access to real- 
time spacecraft and instrument status for 
development of command requests. To facilitate 
communication, MO&CA recommended that the 
audio VOCA (Voice Operations Assembly 
Communications) net be made available to all flight 
team members, keeping both science and operations 
teams informed of current spacecraft status, 

A third communication problem noted by Mars 
Observer MO&CA was the definition of the 
command uplink window, the time period available 
to transmit commands to the spacecraft. Alignment 
of the command requester's requirements, the 
availability of the scheduled commanding windows, 
and the Mission Control Team’s coordination with 
commanding station hand-overs was complex and 
prone to errors. MO&CA recommended that a tool 
be developed to allow the Mission Control Team to 
interpret, implement and verify the command 
requester's requirements for uplink windows. This 
tool was developed by the Mission Control Team 
and is now being used for flight operations. 

Similarly to the Mars Observer Project, the MO&CA 
task was also well received by the 
TOPEX/POSEIDON Project. A MO&CA engineer 
was placed on staff to the TOPEX/POSEIDON Right 
Operations System Manager. The 
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TOPEX/POSEIDON MO&CA task combined 
elements from the VGR and MGN MO&CA 
experience. Like the MGN MO&CA team, 
TOPEX/POSEIDON MO&CA functions as an 
independent unit, and, like VGR MO&CA, 
TOPEX/POSEIDON MO&CA has the ability to 
implement improvements in flight operations 
procedures. 

Once in place on the project in November 1991, 
MO&CA quickly assessed existing flight operations 
plans and noted that an additional process for the 
development and approval of unplanned real-time 
commands was required. MO&CA worked with the 
flight team to define inter-team interfaces for the 
unplanned real-time command process and develop 
the necessary procedures and process descriptions. 
While the flight teams were preparing individual 
team operating procedures, MO&CA was able to 
provide a system level overview and develop the 
additional process and procedures that cross team and 
division boundaries. 

As the project planned to use the real-time command 
process extensively, MO&CA coordinated the 
development of a Real-time Command Library, This 
library consisted of all pre-defined real-time 
command Files developed for repeated utilization 


throughout the life of the mission. The most 
beneficial portion of the Real-time Command Library 
proved to be the Contingency Commands. When 
spacecraft anomalies occurred early in the mission, 
the Contingency Commands facilitated recovery 
operations during a high activity period. The value 
added by the MO&CA Real-time Command Library 
is also visible daily during mission operations. The 
majority of planned real-time commands in the 
TOPEX/POSEIDON Sequence of Events are pulled 
"off-the-shelf 1 from MO&CA s Real-time Command 
Library. 

3. Human Factors Benefits of MO&CA 

MO&CA originated in response to a rise in command 
errors. As was shown in subsequent error analysis 
(Ref. 6) the largest group of errors was human error 
(Figure 2). Thus, addressing human factors in flight 
mission operations has been the overriding benefit of 
MO&CA, The enumeration of these benefits follows. 

The most important of these benefits is a direct 
transfer of knowledge. Originating from the Systems 
Assurance Division at JPL, MO&CA is able to 
transfer knowledge between current missions in 
addition to providing valuable "lessons learned" 
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Hum - Human 
S/W - Software 
Equp - Equipment 
GPR - Guideline/Policy/ 
Requirement 
Doc - Documentation 
Rev - Review Process 
Prod - S/W or Data Product 
Appr - Approval Process 
Form - Forms 
Schd -Schedule 
Other - Other 
Unk - Unknown 
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Figure 2 

Comparison of ISA Analysis - Voyager, Magellan, Mars Observer 
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experience to new flight projects. New projects are 
able to thus benefit directly from both previous and 
ongoing mission operations experience. Lessons 
learned can be incorporated early into project 
requirements, thereby eliminating the amount of 
necessary rework on flight operations procedures. 
The real-time command process and library on the 
TOPEX/POSEIDON project are examples of this 
direct transfer of knowledge. 

Another major benefit is process improvement. 
Process improvement activities require the ability to 
measure and evaluate a process. MO&CA teams 
collect and analyze error data from the ISA reports 
written by flight teams on operational problems. 
Many of MO&CA's recommendations for process 
improvement are based on these reports. This error 
analysis results in improvements not only to the 
project that wrote the report, but also to other flight 
projects via transfer of knowledge. The error analysis 
information is also used for analysis in the overall 
defect prevention/error management program that 
identified human errors as the largest category 
(Figures 1 and 2). 

MO&CA's unique position as an independent unit in 
the flight project organization provides a third major 
benefit to flight projects: the ability to facilitate 
communication and problem solving. Problems that 
span many teams and offices within a flight project 
can be effectively addressed by MO&CA. 
Coordinating real* time command processes is an 
example of this task. Right project members who 
are faced with problems that impact several teams 


often bring the issue directly to the MO&CA 
engineer when they cannot be addressed solely by 
their team. MO&CA is also able to improve the 
efficiency of data reporting that crosses team 
boundaries. On Mars Observer MO&CA worked 
with the teams to eliminate data duplication and 
ensure correct data was reported. 


4. MO&CA and TQM 

The MO&CA function is one example of a Total 
Quality Management (TQM) process at JPL. 
Specifically, MO&CA embodies the TQM principle 
of Continuous Process Improvement (CPI) in which 
processes are continuously examined and analyzed 
for opportunities for improvement. Figure 3 shows 
how MO&CA implements CPI in two ways. First, 
within ongoing projects, the flight mission operations 
environment is established and MO&CA participates 
as a team member. In the course of day-to-day 
operations, anomalies are documented as ISA reports. 
The ISAs then serve as data that is analyzed by 
MO&CA engineers for process improvement 
opportunities. When these opportunities are 
identified, MO&CA provides reports and data to 
support recommendations for improvement to project 
management. Finally, based on management 
approval, MO&CA helps the project implement the 
changes back into the day-to-day mission operations 
environment. This technique was successfully 
implemented on the JPL projects. 
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The second way in which MO&CA implements CPI 
on JPL projects is on new projects or upgrades to 
existing projects. The recommendations that are 
developed from the ISA data analysis on ongoing 
projects are used as input to system requirements on 
new projects. This allows new projects to benefit 
from improvements made on past projects as 
TOPEX/POSEIDON benefited from the experience 
gained on VGR and MGN. 

Using this technique, not only do ongoing projects 
continuously improve, but each new project starts 
with a better set of requirements and better processes 
than the last one. At JPL this continuous 
improvement feedback loop has improved flight 
mission operations processes from the Voyager 
Project, to the Magellan Project, and to the Mars 
Observer and TOPEX/POSEIDON projects. 
Additionally, this Continuous Process Improvement 
reduces both cost and risk of flight mission 
operations. 


5. The Future of MO&CA 

Future flight missions at JPL will have smaller 
spacecraft and flight teams (Refs. 7-8). Development 
times will be reduced and the teams that design and 
build the spacecraft will also staff the flight mission 
operations teams. MO&CA will need to evolve to 
adapt to this changing flight operations environment. 
With smaller flight teams the MO&CA engineer will 
be taking on additional duties such as command 
procedure development and system lead functions, as 
did the engineers on VGR and TOPEX/POSEIDON. 

MO&CA will also participate during the early phases 
of the project, enabling MO&CA to implement 
"lessons learned’ 1 and process improvements during 
development. MO&CA will continue to provide both 
system assurance and engineering assistance to 
operations. MO&CA can assist in developing 
operational procedures and participate in flight team 
training, especially enhancing flight team 
communications and problem solving. This 
participation will streamline procedure development 
and eliminate late changes and upgrades thus 
reducing rework and cost. 

Automation of ISA data tracking and analysis by 
MO&CA will help to make operations process 
monitoring and error analysis more efficient and 
timely. With automation, MO&CA will be able to 
address problem areas quickly. Finally, ISA data 
will be used in a parallel error analysis study. The 
findings of this study (Ref. 6) will enable prevention 
of errors through improved requirements 
development on new projects. 


6. Summary 

JPL flight projects have benefited significantly from 
MO&CA's effort to contain risk and prevent rather 
than rework errors. MO&CA's ability to provide 
direct transfer of knowledge allows new projects to 
benefit from previous and ongoing flight experience. 
The system level view of project operations provided 
by MO&CA enhances communication to facilitate 
problem solving within a flight project. 

MO&CA will continue to evolve to meet flight 
project needs. Early involvement with developing 
projects will ensure that quality is incorporated into 
mission operations during operations development 
and training. 

The MO&CA function at JPL has built quality into 
mission operations, enabling flight teams to operate 
efficiently and effectively in the dynamic flight 
operations environment. Since error analysis has 
shown human error to be the largest error category, 
human factors improvements have, thus far, proved to 
be the major benefit. MO&CA, as a TQM effort, 
focusing on continuous process improvement and 
elimination of rework, will continue to provide 
benefits to flight projects. 
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